Robot perception is far from what humans are capable of. Humans do not only have a complex semantic scene understanding but also extract fine-grained intra-object properties for the salient ones. When humans look at plants, they naturally perceive the plant architecture with its individual leaves and branching system. In this work, we want to advance the granularity in plant understanding for agricultural precision robots. We develop a model to extract fine-grained phenotypic information, such as leaf-, stem-, and vein instances. The underlying dataset RumexLeaves is made publicly available and is the first of its kind with keypoint-guided polyline annotations leading along the line from the lowest stem point along the leaf basal to the leaf apex. Furthermore, we introduce an adapted metric POKS complying with the concept of keypoint-guided polylines. In our experimental evaluation, we provide baseline results for our newly introduced dataset while showcasing the benefits of POKS over OKS.

We present a deep learning framework for wide-field, content-aware estimation of absorption and scattering coefficients of tissues, called Generative Adversarial Network Prediction of Optical Properties (GANPOP). Spatial frequency domain imaging is used to obtain ground-truth optical properties from in vivo human hands, freshly resected human esophagectomy samples and homogeneous tissue phantoms. Images of objects with either flat-field or structured illumination are paired with registered optical property maps and are used to train conditional generative adversarial networks that estimate optical properties from a single input image. We benchmark this approach by comparing GANPOP to a single-snapshot optical property (SSOP) technique, using a normalized mean absolute error (NMAE) metric. In human gastrointestinal specimens, GANPOP estimates both reduced scattering and absorption coefficients at 660 nm from a single 0.2 /mm spatial frequency illumination image with 58% higher accuracy than SSOP. When applied to both in vivo and ex vivo swine tissues, a GANPOP model trained solely on human specimens and phantoms estimates optical properties with approximately 43% improvement over SSOP, indicating adaptability to sample variety. Moreover, we demonstrate that GANPOP estimates optical properties from flat-field illumination images with similar error to SSOP, which requires structured-illumination. Given a training set that appropriately spans the target domain, GANPOP has the potential to enable rapid and accurate wide-field measurements of optical properties, even from conventional imaging systems with flat-field illumination.

Unlike previous works, this open data collection consists of X-ray cone-beam (CB) computed tomography (CT) datasets specifically designed for machine learning applications and high cone-angle artefact reduction. Forty-two walnuts were scanned with a laboratory X-ray set-up to provide not only data from a single object but from a class of objects with natural variability. For each walnut, CB projections on three different source orbits were acquired to provide CB data with different cone angles as well as being able to compute artefact-free, high-quality ground truth images from the combined data that can be used for supervised learning. We provide the complete image reconstruction pipeline: raw projection data, a description of the scanning geometry, pre-processing and reconstruction scripts using open software, and the reconstructed volumes. Due to this, the dataset can not only be used for high cone-angle artefact reduction but also for algorithm development and evaluation for other tasks, such as image reconstruction from limited or sparse-angle (low-dose) scanning, super resolution, or segmentation.

Researchers at Adobe have collaborated with the Beckman Institute for Advanced Science and Technology to develop a new system, based on deep convolutional neural networks, which can extract foreground content from its background intelligently and accurately – and with no need for the blue/green-screen techniques which have dominated cinema for nearly a century. The paper Deep Image Matting outlines the process of evaluating the object which needs to be'clipped' out of its background, which involved the generation of a novel dataset containing 49300 training images intended to accustom algorithms with the challenges of distinguishing backgrounds and eliminating them. Traditional methods of extracting actors or elements from backgrounds, so that they can be inserted into other footage, have always centred around recording the elements (actors, miniatures, etc.) to be extracted in front of a flat field of colour, and relying on photochemical or (later) digital procedures to remove the background. In earlier times, film production workflows generally used blue as a key colour to remove, though Walt Disney studios (which famously took on visual effects work for Alfred Hitchcock's chiller The Birds) used a sodium-based process which keyed on yellow – however, its greater accuracy was offset by the complexity and weight of the equipment required, and the sodium process never gained widespread industry popularity. In the last 15-20 years, green has been adopted as a drop-out colour, since it was proved to be present in less foreground material than blue (for the filming of Superman in the late 1970s, it proved necessary to shoot title actor Christopher Reeve in a costume which was nearer violet than the traditional blue of the man of steel, and to tweak the costume's colour chemically later, so that Reeve did not completely disappear when extracted from a blue background).